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POWER Processor Technology Roadmap 


Enterprise 


- 8 Cores 


- SMT4 


- eDRAM L3 Cache 


1H10 
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POWER7 
45 nm 


POWER7+ 
32 nm 


Enterprise 


- 2.5x Larger L3 cache 
- On-die acceleration 
- Zero-power core idle state 


2H12 


POWERS Family 
22nm 


Enterprise & 
Big Data Optimized 


- Upto 12 Cores 

- SMT8 

- CAPI Acceleration 

- High Bandwidth GPU Attach 


1H14 — 2H16 


POWERS Family __ 
14nm | 


Built for the Cognitive Era 


- Enhanced Core and Chip 
Architecture Optimized for 
Emerging Workloads 


— Processor Family with 
Scale-Up and Scale-Out 
Optimized Silicon 


— Premier Platform for 
Accelerated Computing 


2H17 — 2H18+ 
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POWER8 Processor Family 


Power S8xxLC 


TN C EEn IS 


Enterprise Chip 
Entry SCM 


Single Large Chip 
Up to 12 cores 

Up to 4 socket SMP 
Half memory 

Cost reduced 


NVLINK GPU Enabled 
SuperCompute Node 


SuperCompute Chip 
SC SCM 


Power E850 


Power S8xx/S8xxL 


Scale-Out Chip 
Scale-Out DCM 


Dual Small Chips 


Up to 2 x 6 cores 
VLINK GPU Attach at SMP 


Up to 4 sock 
Up to 48x PCI lanes 
Full Memory 


Power E870/E880 


Enterprise Chip 
Enterprise SCM 


Single Large Chip 

U 0 TO ore 

Up to 16 socket SME 
Up to 32x PCI lanes 
Full Memory 
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Up to four P100 per system—extreme performance; 
OpenPOWER hardware design—leveraging technologies 
from IBM and OpenPOWER partners; 

Focused on high performance applications—advanced 
analytics, Deep Neural Networks, Machine Learning; 
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Nvidia Tesla P100 — First GPU with NVLink 


« Extreme performance—powering HPC, deep leaming, 
and many more GPU Computing areas; 

« NVLink'" —NVIDIA's new high speed, high bandwidth 
interconnectfor maximum application scalability; 

« HBM2—Fastest, high capacity, extremely efficientstacked 
GPU memory architecture; 

« Unified Memory and Compute Preemption—significantly 
improved programming model; 

« 16nmFinFET—enables more features, higher 
performance, and improved power efficiency. 


Up to 2x Application Performance Speedup 
When Next-Gen GPUs Connect via NVLink Versus PCle 
AN 

NVLink significantly increases performance for both us 
GPU-to-GPU communications, and for GPU access 1.75 
to system memory. a 
40 GB/s versus 16 GB/s of PCle Gen3 

5.3 TFLOPS of double precision floating point (FP64) performance eT 

10.6 TFLOPS of single precision (FP32) performance "t 

21.2 TFLOPS of half-precision (FP16)performance © ANSYSFlüent  MultGPUSort — LOCDQUDA  AMBER SD FFT 


3D FFT, ANSYS: 2 GPU configuration, All other apps comparing 4 GPU configuration 
AMBER Cellulose (256x128x128), FFT problem size (2563) 


Up to 3 times the performance of previous generation 
3 times memory bandwidth of K40/M40 


T€— C. NVIDIA Roadmap on POWER 


Kepler — K40, K80 


Pascal — P100 
CUDA 5.5 - 7.0 CUDA 8 
Unified Memory 
Kepler 


POWER8 


POWERS NVLink 
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Volta 
CUDA9 


VLink 2.0 


wes 2 POWERS Family - Deep Workload Optimizations 22 
Emerging Analytics, Al, Cognitive 
- New core for stronger thread performance sang 


€ | torch 
- Delivers 2x compute resource per socket us 
- Built for acceleration - OpenPOWER solution enablement DB2 BLU Soa Caffe 


Technical / HPC 
- Highest bandwidth GPU attach OAK RIDGE 
- Advanced GPU/CPU interaction and memory sharing panes E Lawrence Livermore 


- High bandwidth direct attach memory National Laboratory 


Cloud / HSDC 
- Power / Packaging / Cost optimizations for a range of platforms 
- Superior virtualization features: security, power management, QoS, interrupt 


© 


- State of the art IO technology for network and storage performance rackspace 
Enterprise 
- Large, flat, Scale-Up Systems - | a 
- Buffered memory for maximum capacity rower 
- Leading RAS 1 
ORACLE 


- Improved caching 
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New Core Microarchitecture 
Stronger thread performance 
Efficient agile pipeline 
POWER ISA v3.0 


Enhanced Cache Hierarchy 
120MB NUCAL3 architecture 
12 x 20-way associative regions 
Advanced replacement policies 
Fed by 7 TB/s on-chip bandwidth 


Cloud + Virtualization Innovation 
Quality of service assists 

New interrupt architecture 

Workload optimized frequency 
Hardware enforced trusted execution 
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POWERS Processor — Common Features HE: 


SMP/Accelerator Signaling Memory Signaling 


SMP Signaling 
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SMP/Accelerator Signaling Memory Signaling 


14nm finFET Semiconductor Process 


* Improved device performance and 
reduced energy 


* 17 layer metal stack and eDRAM 
* 8.0 billion transistors 


Leadership 
Hardware Acceleration Platform 
Enhanced on-chip acceleration 


Nvidia NVLink 2.0: High bandwidth 
and advanced new features (25G) 
CAPI 2.0: Coherent accelerator and 
storage attach (PCle G4) 


* NewCAPI: Improved latency and 
bandwidth, open interface (25G) 


State of the Art I/O Subsystem 
e PCle Gen4 — 48 lanes 


High Bandwidth 
Signaling Technology 


* 16 Gb/s interface 
— Local SMP 

e 25 Gb/s Common Link interface 
— Accelerator, remote SMP 


nn © POWERS Processor Family 


Four targeted implementations Pole CORNE AZE 


SMT4 Core SMT8 Core 
u 24 SMT4 Cores / Chip 12 SMT8 Cores / Chip 
SMP scalability / Memory subsystem Linux Ecosystem Optimized PowerVM Ecosystem Continuity 


Scale-Out - 2 Socket Optimized 
Robust 2 socket SMP system 
Direct Me mory Attach Cache and Interconnect 


* Up to 8 DDR4 ports - 
* Commodity packaging form factor |: El F | sp ER 
CD GE ED CÓ (ome (ec) 


Scale-Up - Multi-Socket Optimized 
Scalable System Topology / Capacity 
* Large multi-socket Cache and Interconnect Cache and Interconnect 


Buffered Memory Attach 
HEB 
DERO 


e 8 Buffered channels 
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pierce New POWERS Cores 


Optimized for Stronger Thread Performance and Efficiency 
* Increased execution bandwidth efficiency for a range of workloads including commercial, cognitive and analytics 
e Sophisticated instruction scheduling and branch prediction for unoptimized applications and interpretive languages 
e Adaptive features for improved efficiency and performance especially in lower memory bandwidth systems 


Available with SMT8 or SMT4 Cores 
8 or 4 threaded core built from modular execution slices 


POWER9 SMT4 Core 

* Linux Ecosystem Focus 
* Core Count/ Socket 

e Virtualization Granularity 


POWER? SMT8 Core 

* PowerVM Ecosystem Continuity 
e Strongest Thread 

* Optimized for Large Partitions 


DW DW DW DW 
LSU LSU LSU LSU 


SMT8 Core 


SMT4 Core 
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rowersystems> POWERS Core Execution Slice Microarchitecture 


Modular Execution Slices 
4 x 128b 2 x 128b 128b 64b 


BUM Super-slice Super-slice Super-slice Slice 
p "de" 


POWERS SMT8 Core POWER9 SMT8 Core POWER9 SMT4 Core 


Re-factored Core Provides Improved Efficiency & Workload Alignment 
e Enhanced pipeline efficiency with modular execution and intelligent pipeline control 


* Increased pipeline utilization with symmetric data-type engines: Fixed, Float, 128b, SIMD 
e Shared compute resource optimizes data-type interchange 
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POWERS Core Pipeline Efficiency IER 


Shorter Pipelines with Reduced Disruption 


Improved application performance for modern codes 
* Shorten fetch to compute by 5 cycles 
* Advanced branch prediction 


Higher performance and pipeline utilization 
* Improved instruction management 
— Removed instruction grouping and reduced cracking 
— Enhancedinstruction fusion 
— Complete up to 128 (64 — SMT4 Core) instructions per cycle 


Reduced latency and improved scalability 


e Local pipe control of load/store operations 
— |mproved hazard avoidance 
— Local recycles — reduced hazard disruption 
— Improved lock management 
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POWERS Pipeline 


Fetch to Compute 
Reduced by 5 cycles 


POWERS Pipeline 
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SMT4 Core Resources 


Fetch / Branch 

e 32kB, 8-way Instruction Cache 
* 8 fetch, 6 decode 

* 1x branch execution 


Slices issue VSU and AGEN 
e 4x scalar-64b / 2x vector-128b 
e 4x load/store AGEN 


Vector Scalar Unit (VSU) Pipes 
4x ALU + Simple (64b) 
4x FP + FX-MUL + Complex (64b) 
2x Permute (128b) 
2x Quad Fixed (128b) 
2x Fixed Divide (64b) 
1x Quad FP & Decimal FP 
1x Cryptography 


Load Store Unit (LSU) Slices 
e 32kB, 8-way Data Cache 
* Up to 4 DW load or store 
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POWERS - Core Compute 


Symmetric Engines Per Data-Type for 
Higher Performance on Diverse Workloads 


us ce SMT4 Core 
Branch ; : Instruction / lop 
Dispatch: Allocate / Rename Completion Table 
x6 


128b 
Super-slice 


Efficient Cores Deliver 2x Compute Resource per Socket 
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circ POWER ISA v3.0 


New Instruction Set Architecture Implemented on POWER9 


Broader data type support 

e 128-bit IEEE 754 Quad-Precision Float — Full width quad-precision for financial and security applications 
* Expanded BCD and 128b Decimal Integer — For database and native analytics 

* Half-Precision Float Conversion — Optimized for accelerator bandwidth and data exchange 


Support Emerging Algorithms 
e Enhanced Arithmetic and SIMD 


e Random Number Generation Instruction » OW m R 


Accelerate Emerging Workloads 
* Memory Atomics — For high scale data-centric applications 
* Hardware Assisted Garbage Collection — Optimize response time of interpretive languages 


Cloud Optimization 

* Enhanced Translation Architecture — Optimized for Linux 

* New Interrupt Architecture — Automated partition routing for extreme virtualization 
e Enhanced Accelerator Virtualization 

e Hardware Enforced Trusted Execution 


Energy & Frequency Management 
e POWERO9 Workload Optimized Frequency — Manage energy between threads and cores with reduced wakeup latency 
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CONO POWER zn Data Capacity & Throughput PER 


Big Caches for Massively Parallel Compute Extreme Switching Bandwidth for the 
and Heterogeneous Interaction Most Demanding Compute and Accelerated Workloads 
L3 Cache: 120 MB Shared Capacity NUCA Cache High-Throughput On-Chip Fabric 
e 10MB Capacity + 512k L2 per SMT8 Core e Over 7 TB/s On-chip Switch 
* Enhanced Replacement with Reuse & Data-Type Awareness + Move Data in/out at 256 GB/s per SMT8 Core 


12 x 20 way associativity 


POWER9 17 Layers of Metal 


E NEN pe————————————»————LEEEÉLZZÉSSLSZZZEELEEZIZEEÉLEZÉE UC EL LLÉZLDZIJZELZLZZZZZLEZZZLEZEZLAÉAAAAM 
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256 GB/s x 12 


N 
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IBM & Ut IBM & 
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NR POWERS - Dual Memory Subsystems 


Scale Out Scale Up 
Direct Attach Memory Buffered Memory 


EX 


ttt tt ttt 


8 Direct DDR4 Ports 8 Buffered Channels 
e Upto 120 GB/s of sustained bandwidth e Upto 230GB/s of sustained bandwidth 
e Low latency access e Extreme capacity — up to 8TB / socket 
e Commodity packaging form factor e Superior RAS with chip kill and lane sparing 
* Adaptive 64B / 128B reads e Compatible with POWER system memory 


e Agnostic interface for alternate memory innovations 


€ 2016 IBM Corporation 16 


CONO 
Modular Constructs > High-speed 25 Gb/s Signaling 


MP interconnect 


putti-prawe" P 


NVLINK 2 GPU Accelerator Attach 


Open 
WAP IAccelerator pe 


Utilize Best-of-Breed | : " 


25 Gb/s Optical-Style Flexible & Modular 
: i Packaging 
Signaling Technology ee 
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New External Interrupt Virtualization Engine (XIVE) UE 


- Prior processors distributed interrupts across system 
> Significant Software overhead to route interrupts 
- New XIVE hardware routes correctly 15! time 
> Eliminates host processor overhead 
> Directly target guest Operating System 
> Enable User level Interrupt 


Interrupts 
Network ji— 


Inactive 
Virt.Proc 


© 2016 IBM Corporation 


Power systems D 


© 2016 IBM Corporation 


On-Processor Accelerators 


Platform Virtualization Features: Accelerators 


Virtualized: User mode invocation (No Hypervisor Calls) 
Industry Standard GZIP Compression / Decompression 
AES Cryptography Support 

True Random Number Generation 

Data Mover 
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Open Innovation Interfaces: Open CAPI 


CAPI 1.0, 2.0 Architecture New CAPI Architecture 


DE 


POWER-specific 
coherence IP 


POWER-specific 
coherence IP 


Architected 
Programming 


POWER-specific } Interface 


coherence IP y 
^ 


| Architected 
Programming 
Interface 


Accelerated 
Application 


Attached CAPI-Accelerated Chip Attached CAPI-Accelerated Chip 


Open Industry Coherent Attach 
- Latency / Bandwidth Improvement 
- Removes Overhead from Attach Silicon 
- Eliminates "Von-Neumann Bottleneck" 
- FPGA / Parallel Compute Optimized 
- Network/Memory/Storage Innovation 
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OpenCAPI Approach ( CopencaPi 


e What is OpenCAPI? 
e OpenCAPI is an Open Interface Architecture that allows any microprocessor to attach to 
e Coherent user-level accelerators and I/O devices 
* Advanced memories accessible via read/write or user-level DMA semantics 
* Agnostic to processor architecture 
* Key Attributes of OpenCAPI 
* High-bandwidth, low latency interface optimized to enable streamlined implementation of attached devices 


e 25Gbit/sec signaling and protocol built to enable very low latency interface on CPU and attached device 


* Complexities of coherence and virtual addressing implemented on host microprocessor to simplify attached devices and 
facilitate interoperability across multiple CPU architectures 


* Attached devices operate natively within an application's user space and coherently with processors 
* Allows attached device to fully participate in application without kernel involvement/overhead 

* Supports a wide range of use cases and access semantics 
* Hardware accelerators 
* High-performance 1/0 devices 


* Advanced memories 


21 e 100% Open Consortium / All company participants welcome / All ISA participants welcome 
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OpenCAPI 3.0 Features ( CopencaPi 


Base Accelerator Support 


* Accelerator Reads with no intent to cache, DMA write using Program Addresses 


The accelerator is working in the same address domain as the host application 


Pointer chasing, link lists are all now possible without Device Driver involvement 
Address translation on host (processor) with error response back to the accelerator 


* Very efficient translation latency mechanism using host processor Address Translation Cache (ATC) and MMU 
e Non-posted writes only 


* Ability for Partial Read/Write DMAs 
* Write with byte enables 


* Translate touch to warm up address translation caches 


Allows accelerator to reduce translation latency when using a new page 
* Wake Up host thread 


Very efficientlow latency mechanism in lieu of either interrupts or host processor polling mechanism of memory 
* Atomic Memory Operations (AMO) to Host Processor Memory 


Accelerator can now perform atomic operations inthe same coherent domain just like any other host processor thread 
22 
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Virtual Addressing (OpenCAPI 


* An OpenCAPI device operates in the virtual address spaces of the applications that it supports 


* Eliminates kernel and device driver software overhead 


* Improves accelerator performance 
* Allows device to operate directly on application memory without kernel-level data copies or pinned pages 


* Simplifies programming effort to integrate accelerators into applications 


* The Virtual-to-Physical Address Translation occurs in the host CPU 


* Reduces design complexity of OpenCAPl-attached devices 
* Makes it easier to ensure interoperability between an OpenCAPI device and multiple CPU architectures 


* Since the OpenCAPI device never has access to a physical address, this eliminates the possibility of a 
defective or malicious device accessing memory locations belonging to the kernel or other applications that 


it is not authorized to access 
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OpenCAPI 3.0 Features (cont.) ( COpencAPI 


Base Accelerator Support 
* MMIO slave 


e Accelerators have BAR space that provide MMIO register capability 
* Configuration space facility slave 


* Efficient discovery and enumeration of accelerators 
* OpenCAPI attached Memory 


e High bandwidth and low latency memory home agent capability 

e  Load/Store model access to OpenCAPI attached memory 

* Host Application can access memory attached to OpenCAPI endpoint as part of coherent domain 
e Data resides very close to the consumer with very low latency 


* Atomic Memory Operations (AMO) support toward OpenCAPI attached memory 
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© POWER9 - CPU Core Performance 


Power Systems `” 


Socket Performance 


Graph Business 
Analytics Intelligence 


POWERS Commercial Scripting 
Baseline 


Scale-Out configuration @ constant frequency 
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—PÓÀ POWER? - Premier Acceleration Platform PEE 


* Extreme Processor / Accelerator Bandwidth and Reduced Latency 
* Coherent Memory and Virtual Addressing Capability for all Accelerators POWER9 
* OpenPOWER Community Enablement — Robust Accelerated Compute Options 


DMI 
e State of the Art I/O and Acceleration Attachment Signalin 
— PCle Gen 4 x 48 lanes — 192 GB/s duplex bandwidth 
— 25Gb/s Common Link x 48 lanes — 300 GB/s duplex bandwidth 


* Robust Accelerated Compute Options with OPEN standards 
— On-Chip Acceleration — Gzip x1, 842 Compression x2, AES/SHA x2 


]oeuuooJeju| pue euoe2 
S902) 10SS89014 


— CAPI 2.0 — 4x bandwidth of POWERS using PCIe Gen 4 Toe 
NVLink 2.0 — Next generation of GPU/CPU bandwidth and integration using 25G 
Accel 


Open CAPI 3.0 — High bandwidth, low latency and open interface using 25G 
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nn POWERS - Ideal for Acceleration 


Extreme CPU/Accelerator Bandwidth 


PCle Gen3 x16 PCle Gen4 x16 POWERS with NVLink 1.0 POWERSO with 25G Link 


| cpu |Q 
1x 


Increased Performance / Features / Acceleration Opportunity 


Seamless CPU/Accelerator Interaction Broader Application of Heterogeneous Compute 
e Coherent memory sharing e Designed for efficient programming models 
e Enhanced virtual address translation e Accelerate complex analytic / cognitive applications 


» Data interaction with reduced SW & HW overhead 


€ 2016 IBM Corporation 27 


TETS POWER - Ecosystem Enablement Hi 


©) OpenPOWER* 6 
P ™ ms t Ns e ae mm, 
p e n BSU Yo IS WER CAMBRIDGE Carnegi rie e Hartree Centre fi) we. aa i s 
ARIZONA STATE asmi a See fano ES O CARES Mellon CINECA QA / mtb e) g m I 
UNIVERSITY ial and Research b University GSC mem uod 

= on iB — 

F oun d ati on f)jüLICH [E Lawrence Livermore Yen kc} Ve OAK (SU @ d 

FORSCHUNGSTENTRUM National Laboratory (9) NUS Lae as "RIDGE Oregon t 


UNIVERSITY 


NÉ 


‘ational Laboratory 


Sandia ANS À Symbi sis si fe d 
. . A mg SASTRA mm SDSC 4p? @ Gc 
* Accelerating Open Innovation °C a= immen ui ~ F o & e 
cam Quid a ^. A = 
e Grown from 5 to over 200 A Implementation / HPC /Research  ::: 9) re a & RRA 
members in less than 3 years TT à e : ane "oae F 
H Software A ees rer EE A. Google WM quo fa redis Tai ubuntu 
H (AN ay, í TRES 
POWERS: Engineered for MM. © S" Gil scara aou. CSPI DRE E4 (E) IEH inspurss € Marl 
H i ÆMicroway NEC mmm O LL LLL B lllE 
OpenPOWER Application 47 ME NP CUL — a - 
System / Integration Il SES COMPUTING ENGINEERING ' | Bko CQ? SAME 
* Built for a Broad Range of | í . 
Deployments and Platforms E ju AVAGO Gewe, e Bon — Cheb Mera E ici YE 
* Open and Flexible Solutions HGST HITACHI XXiphi sese, MAXERER AA Pau (Micron Microsemi — MYaicoM  ANallateh © 
meteo Oe M tomate Mellanox aus NVIDIA. 
e A 7 .. 
Ideal for Developers E I/O / Storage / Acceleration occ «EP Samisk wm Ke 99 9  gXUNX 
to D Aan IBH men | Ge 6 
i ards / Systems Celestica. Maran LEP nventec mesi £n TYAN wistron 
E Chip /SOC PHÓ OIDT Cinfineon 4» POWERCORE (suc 
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piers S POWE R9 Processor 


Built for the Cognitive Era jopenPowER: 


Enhanced Core and Chip Architecture for Emerging Workloads 
* New Core Optimized for Emerging Algorithms to Interpret and Reason 
e Bandwidth, Scale, and Capacity, to Ingest and Analyze 


Processor Family with Scale-Out and Scale-Up Optimized Silicon 
e Enabling a Range of Platform Optimizations — from HSDC Clusters to Enterprise Class Systems 
* Extreme Virtualization Capabilities for the Cloud 


Premier Acceleration Platform 

* Heterogeneous Compute Options to Enable New Application Paradigms 
e State of the Art I/O 

* Engineered to be Open 
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Special notices 


This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in other 
countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM offerings available in 
yourarea. 


Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions on the 
capabilities of non-IBM products should be addressed to the suppliers of those products. 


IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does notgive you any 
license to these patents. Send license inquiries, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY 10504-1785 
USA. 

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. 

The information contained in this document has notbeen submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either 
expressed orimplied. 

All examples cited ordescribed in this document are presented as illustrations of the mannerin which some IBM products can be used and theresults 
that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions. 
IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions worldwide to 
qualified commercial and governmentclients. Rates are based ona client's credit rating, financing terms, offering type, equipment type and options, and 
may vary by country. Otherrestrictions may apply. Rates and offerings are subject to change, extensionor withdrawal without notice. 

IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies. 

All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary. 

IBM hardware products are manufactured from new parts, ornew and serviceable used parts. Regardless, our warranty terms apply. 

Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are dependent on 
many factors including system hardware configuration and software design and configuration. Some measurements quoted in this document may have 
been made on development-level systems. There is no guarantee these measurements will be the same on generally-available systems. Some 
measurements quoted in this document may have been estimated through extrapolation. Users of this document should verify the applicable datafortheir 


specific environment. 
Revised September 26, 2006 
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Special notices (continued) 


IBM, the IBM logo, ibm.com AIX, AIX (logo), IBM Watson, DB2 Universal Database, POWER, PowerLinux, PowerVM, PowerVM (logo), PowerHA, Power Architecture, Power Family, POWER Hypervisa, 
Power Systems, Power Systems (logo), POWER2, POWER3, POWER4, POWER4+, POWERS, POWER5+, POWER6, POWER6+, POWER7, POWER7+, and POWERS are trademarks or registered 
trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information 
with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered 
or common law trademarks in other countries. 


A full list of U.S. trademarks owned by IBM may be found at: http:/www.ibm.com/legal/copytrade.shtml. 


NVIDIA, the NVIDIA logo, and NVLink are trademarks or registered trademarks of NVIDIA Corporation in the United States and other countries. 


Linux is aregistered trademark of Linus Torvalds in the United States, other countries or both. 
PowerLinux™ uses the registered trademark Linux® pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the Linux® mark on a world-wide basis. 


The Power Architecture and Power.org wordmarks and the Power and Power.org logos and related marks are trademarks and service marks licensed by Power.org. 
The OpenPOWER word mark and the OpenPOWER Logo mark, and related marks, are trademarks and service marks licensed by OpenPOWER. 


Other company, product and service names may be trademarks or service marks of others. 
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